Papers with wordpiece tokenization
CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters (2020.coling-main)
Copied to clipboard
Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, Jun’ichi Tsujii
| Challenge: | Pre-trained language representations from Transformers have become the most popular choice for building NLP systems. |
| Approach: | They propose a new variant of BERT that drops the wordpiece tokenization system altogether . they propose 'characterBERT' module to represent entire words by consulting their characters . |
| Outcome: | The proposed model improves performance on a variety of medical domain tasks while producing robust, word-level, and open-vocabulary representations. |